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I PAPER I 

Strongly Secure Privacy Amplification Cannot Be 
Obtained by Encoder of Slepian-Wolf Code* 

Shun WATANABE^^), Ryutaroh MATSUMOTO^b), and Tomohiko UYEMATSU+t^), Members 



SUMMARY The privacy amplification is a technique to dis- 
till a secret key from a random variable by a function so that the 
distilled key and eavesdropper's random variable are statistically 
independent. There are three kinds of security criteria for the key 
distilled by the privacy amplification; the normalized divergence 
criterion, which is also known as the weak security criterion, the 
variational distance criterion, and the divergence criterion, which 
is also known as the strong security criterion. As a technique to 
distill a secret key, it is known that the encoder of a Slepian-Wolf 
(the source coding with full side-information at the decoder) code 
can be used as a function for the privacy amplification if we em- 
ploy the weak security criterion. In this paper, we show that the 
encoder of a Slepian-Wolf code cannot be used as a function for 
the privacy amplification if we employ the criteria other than the 
weak one. 

key words: privacy amplification, secret key agreement, 
Slepian- Wolf coding, strong security, variational distance, weak 
security 

1. Introduction 

One of the hmdamental problems in the cryptography 
is the key agreement in which the legitimate parties, 
usually referred to as Alice and Bob, share a secret 
key that is not known by the eavesdropper, usually re- 
ferred to as Eve. The problems on the key agreement 
in the information theory was initiated by Maurer [11], 
and was also studied by Ahlswede and Csiszar [1]. In 
their formulation, Alice, Bob, and Eve have correlated 
random variables X", F", and Z" respectively. Then, 
Alice and Bob generate a secret key from (A", F") by 
using the public (authenticated) communication. 

Typically, a key agreement protocol consists of two 
procedures: the information reconciliation [3], [4] and 
the privacy amplification [2] , [3] . The purpose of the 
information reconciliation for Alice and Bob is to share 
an identical random variable (with high probability) 
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by using the public communication. The privacy am- 
plification is a technique to distill a secret key from 
the shared random variable by using a function so that 
Eve's knowledge and the secret key are statistically in- 
dependent. In order to focus on the privacy amplifica- 
tion, we assume that Alice and Bob initially share the 
random variables A" = in the rest of this paper. 

As the security of the secret key distilled by the 
privacy amplification, there are three kinds of security 
criteria: the normalized divergence criterion, which is 
also known as the weak security criterion, the varia- 
tional distance criterion, and the divergence criterion, 
which is also known as the strong security criterion. 
The normalized divergence criterion requires that the 
key and Eve's knowledge is (almost) statistically in- 
dependent in the sense that the divergence divided by 
n (normalized divergence), or equivalently the mutual 
information divided by n, is negligible. On the other 
hand, the variational distance criterion and the diver- 
gence criterion require that the key and Eve's knowl- 
edge is (almost) statistically independent in the sense 
of the variational distance and the divergence are neg- 
ligible respectively. 

Traditionally, the normalized divergence criterion 
was employed in the study of the key agreement 
(e.g. [1],[11]). However, as Maurer and Wolf pointed 
out [12], Eve might know a large part of the key even 
if the key satisfies the normalized divergence criterion. 
Therefore, we should use the divergence criterion. In- 
deed, recent studies on the key agreement employ the 
divergence criterion (e.g. [7], [15]). 

As one of techniques to distill a secret key, it is 
known that the encoder of a Slepian-Wolf (the source 
coding with full side-information at the decoder) code 
[19] can be used as a function for the privacy amplifi- 
cation. For example, Ahlswede and Csiszar used this 
technique implicitly [1] , and Muramatsu used this tech- 
nique explicitly [14]. 

To describe the technique more precisely, let us 
consider the Slepian-Wolf code system such that A" 
is the principal source and Z" is the side-information. 
Then, the output of the encoder, which is regarded as 
the key, satisfies the normalized divergence criterion if 
the coding rate of the code is close to the compression 
limit and the decoding error probability of the code is 
negligible. However, it has not been clarified whether 
this technique can be used for the divergence criterion. 
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In this paper, we show that above mentioned tech- 
nique cannot be used for the divergence criterion. Ac- 
tually, we show that the divergence grows infinitely in 
the order of -^/n, which suggests that Eve might know 
a large part of the key. 

Although the divergence criterion is the strongest 
notion of security among the above mentioned three 
criteria, some researchers (eg. [18]) deem that the vari- 
ational distance criterion is appropriate notion of secu- 
rity because it matches with the universally compos- 
able security [5], which requires that the actual distri- 
bution of the key and Eve's knowledge is indistinguish- 
able from the ideal distribution with which the key is 
uniformly distributed and independent of Eve's knowl- 
edge. Therefore, it is worthwhile clarifying whether the 
key obtained by the above mentioned technique satis- 
fies the variational distance criterion or not. In this 
paper, we show that the key obtained by the technique 
does not satisfy the variational distance criterion. Ac- 
tually, we show that the variational distance converges 
to one (the maximum amount), which means that the 
actual distribution and the ideal distribution is com- 
pletely distinguishable. 

The results in this paper are also interesting from 
the view point other than the privacy amplification. 
The above mentioned technique can be regarded as the 
Slepian-Wolf version of the folklore theorem shown by 
Han [9]. Recently, Hayashi [10] showed that the folk- 
lore theorem does not hold if we employ the variational 
distance criterion nor the divergence criterion instead 
of the normalized divergence criterion. Our results can 
be regarded as a generalization of Hayashi's results for 
the Slepian-Wolf code. 

The rest of this paper is organized as follows: In 
Section 2, we review the basic notations, the privacy 
amplification, and the above mentioned technique. In 
Section 3, we show our main results concerning the di- 
vergence criterion and their proofs. In Section 4, we 
show our main results concerning the variational dis- 
tance criterion and their proofs. In Section 5, we con- 
clude the paper. 

Finally, it should be noted that the results on the 
divergence criterion and the variational distance crite- 
rion cannot be derived from each other, though a weak 
version of the result on the divergence criterion, i.e., 
the fact that the divergence does not converge to zero 
(Corollary 12). can be derived as a corollary of the re- 
sults on the variational distance criterion. The weak 
version only suggests that Eve might know a few bits 
about the key whose length grows infinitely as n goes 
to infinite, which is not a serious problem in practice. 
On the other hand, the result in Section 3 suggests that 
Eve's knowledge about the key also grows infinitely as 
the length of the key goes to infinite, which is a seri- 
ous problem in practice. Therefore, we need to treat 
both the divergence criterion and the variational dis- 
tance criterion separately. 



2. Preliminaries 

2.1 Privacy Amplification 

In this section, we review the basic notations related 
to the privacy amplification. Suppose that Alice and 
Bob have a random variable X" on A"", and Eve has 
a random variable Z" on Z", where (X", Z") are in- 
dependently identically distributed (i.i.d.) according 
to the probability distribution Pxz- In this paper, we 
assume that X and Z are finite sets. 

The privacy amplification [2] , [3] is a technique to 
distill a secret key 5'„ from X" by using a function 

/„:A'"^X„ = {1,...,M„} 

so that the key and Eve's information Z" are statisti- 
cally independent and the key is uniformly distributed 
on the key alphabet Mn - The joint probability distri- 
bution of the key and Eve's information is given by 

PS^Z^{S,7:')= ^X"Z"(X",Z") (1) 

x"e/,7'(.s) 

for (s, z") S A4n X Z", where we defined fn^{s) = 
{.T": /„(a;") = s}. 

For probability distributions P and Q on A, let 

be the variational distance (divided by 2), and let 
i.(P[|Q) = |:P(a)log^ 

be the divergence respectively [6], where we take the 
base of the logarithm to be e throughout the paper. By 
using these two quantities, we introduce three kinds of 
security criteria on the privacy amplification. 

Definition 1 If a sequence of functions {/,,} satisfies 

lim -D{f„) = (2) 

n — >oo 77, 

for 

D{U) ^ D{Ps^z4Pu^ X Pz^), 

then we define the privacy amplification by {/«} to be 
secure with respect to the normalized divergence crite- 
rion, where Pu^ is the uniform distribution on Mn- 

Definition 2 If a sequence of functions {/„} satisfies 
lim A(/„) = (3) 

n — >-oo 

for 

A(/„) =d(Ps„Z",Pt/ X Pz,0, 

then wc define the privacy amplification by {/«} to be 
secure with respect to the variational distance criterion. 
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Definition 3 If a sequence of functions {/«} satisfies 
lim D{U) = 0, (4) 

n — ^oo 

then we define the privacy ampUfication by {/„} to be 
secure with respect to the divergence criterion. 

We can show that Eq. (4) impUes Eq. (3) by using 
Pinsker's inequaUty [6]. We can also show that Eq. (3) 
impUes Eq. (2) by using [7, Lemma 1]. 

The security criteria in Definitions I and 3 are 
equivalent to the weak security criterion and the strong 
security criterion defined in [12]. The security criterion 
in Definition 2 is widely used recently (eg. [18]) because 
it match with the universally composable security [5], 
which requires that the actual distribution Ps„Z" and 
the ideal distribution Pu^ x P^n are indistinguishable. 
Although the divergence is also related to the distin- 
guishability between distributions, the variational dis- 
tance is directly related to the distinguishability be- 
cause the optimized average probability of the correct 
discrimination is given by 



- max [Ps„Z"(^) + /^c/„Z"(-4^)] 

= lll + diPs„Z-,Pu„ X Pz")], 



(5) 



which is a straightforward consequence of the definition 
of the variational distance [6] , where the superscript c 
designate the complement of the set. 

2.2 Privacy Amplification by an Encoder of Slepian- 
Wolf Code 

In this section, we explain the Slepian-Wolf code, and 
then review the relation between the privacy amplifi- 
cation and the Slepian-Wolf code. We consider the 
Slepian-Wolf code system in which AT" is the princi- 
pal source and is the side-information. The code 
system consists of the encoder 

: A-" ^ Mn 

and the decoder 

and we denote the code as $„ = [ipmi^n)- The error 
probability of the code is defined as 

= Px"Z"({(x", z") : Vn(</'n(x"),z") ^ a;"}). 

For any real number P > 0, we call the rate R is 
achievable if there exists a sequence of codes {$,1} that 
satisfies 

hm sup — log Af„ < P and hm £($„) = 0. 
Then, we define the compression limit as 



Rf{X\Z)= mi{R : R is achievable}. 

It is well known that the compression limit coincide 
with the conditional entropy [19], i.e., P/(A'|Z) = 
HiX\Z). 

If a sequence of code {^n} satisfies 



lim — log A/„ 

n — >oo ri 



H{X\Z) 



and 



lim e(<i>„ 



0, 



(6) 



(7) 



then we call the sequence of codes compression 
limit achieving codes. When {$„} satisfies Eq. (6), it 
should be noted that the error probability depends on 
the second order rate ^ log ^„hIx\z) ■ For later use, 
we present the converse coding theorem concerning the 
tradeoff between the error probability and the second 
order rate. The theorem is a Slepian-Wolf coding ver- 
sion of the result on the second order asymptotic of the 
source coding [10]. 

Theorem 4 Let 6 G K be arbitrary real number. For 
any code sequence {'I'n}; if the error probability satis- 
fies 



limsupe((f>„) < 1 - G 

n — ^00 

then the rate satisfies 
r • f 1 1 ^'^^ 



(8) 
(9) 



where 
G{t) 



/27r 



-^"'^du 



is the cumulative distribution function of the Gaussian 
distribution with mean and variance 1, and where we 
set 

1 



Var 



log- 



x\z 



{X\Z) 



(10) 



This theorem is a straight forward consequence of the 
central limit theorem, and we show a proof in Appendix 
A. 

The following proposition states that the encoders 
of compression limit achieving codes can be used as 
functions for the privacy amplification if we employ the 
normalized divergence criterion. 

Proposition 5 If a sequence of code {$„ — ((^„, -!/;„)} 
satisfies Eqs. (6) and (7), then we have 

1 



lim -D{(j)n) 

n— ^00 71 



0. 



This proposition can be proved almost in a similar man- 
ner to [14, Theorem 1]. Note that Proposition 5 can be 
regarded as the Slepian-Wolf version of the folklore the- 
orem [9] (sec also [8, Theorem 2.6.4]). 
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3. Divergence Criterion 

3.1 Statement of Results 

In this section, we sliow our main results concerning the 
divergence criterion, which are proved in Section 3.2. In 
Section 2.2, we showed that the encoders of compres- 
sion limit achieving codes can be used as functions for 
the privacy amplification which is secure in the sense 
of the normalized divergence criterion. The following 
theorem states that the divergence actually grows in- 
finitely in the order of ^/n (Eq. (12)), which suggest 
that Eve might know a large part of the key. The fol- 
lowing theorem can be regarded as a generalization of 
[10, Theorem 8] for the Slepian-Wolf code. 

Theorem 6 Suppose that a sequence of functions, /„ : 
A"" {1, . . . , Af„} for n 1, 2, . . ., satisfies Eq. (6), 
and let 

h = lim inf —= log ■ 



n^oo ^ ^ pnH{X\Z) ' 

Then, we have 



limM ^D{fn)> I {b ~ au)g{u)du, (11) 



where 



is the density function of the Gaussian distribution with 
mean and variance 1, and where cr^ is the variance 
defined in Eq. (10). 

Suppose that the Slepian-Wolf code sequence 
{<!>„ = {(prijipn)} satisfies Eq. (6) and 

hm sup £(<!>„) < 1. 

n — >oo 

Then, there exist a real number 6 G K such that 



limsupe(<I>„) < 1 - G - 



Since the right hand side of Eq. (11) is an increasing 
function of b and is positive for any 6 G M, Theorem 
4 and Theorem 6 imply that there exists a constant 
K > and an integer uq such that 



for every n > uq. 

3.2 Proof of Theorem 6 



(12) 



Lemma 7 Let 



T„= (x",z"): 



flog \ — ; — 7 -nH(X\Z)] < - 



Then, we have 

+Px'^Z'^ [log M„ - log Px^ {%:)] . 
Proof. Let 

and let : X"- x Z" —* be the function defined by 
" /„(a;") if (x",z") ^T;, 



if (x",z") gt;, 



We set the random variable S'^ = /;(X",Z"). Then, 
we have 

V Px"Z"(a;",z")log ^, , , 

E ^s,zAs,^nioE^^;^^. 

(13) 

By using the log-sum inequality [6], we can upper 
bound the last term in Eq. (13) as 



PzA^") 

PS'^Z'^{S,Z") 



< Px"Z"(7;n[logA/„ - logFx"Z"(7;n]. (14) 
Let /4' : A^^ be the function defined by 

s if ,s e Mn 



Then, we have S'„ — fn{S'^). Since the conditional en- 
tropy does not increase by a function [6] , by combining 
Eqs. (13) and (14), we have the assertion of the lemma. 

□ 



In order to show a proof of Theorem 6, we need the 
following lemma. 



Proof of Theorem 6 

By using Lemma 7, we have 
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--D{fn) 



> P.Y"Z"(r„)-^log-;^ 



{H{S^\Z")-nH{X\Z)) 



\z) 

-a Px^zA^",^'") 
{x",z")eT„ 

[log ■ - 72H(X\Z) 

+ 1 P^„^„ (T„'=) log Px^z^ [r:). 



By taking the limit of both side and using the central 
limit theorem with respect to the cumulative distribu- 
tion function 



1 



log- 



1 



-nB{X\Z) < It ^ , 



we have 

lim inf ^D(/„) 

f h\ ~ 
>hG\ — \ — a / ug{u)du 

J J~oo 

{b — au)g{u)du, 
which completes the proof □ 
4. Variational Distance Criterion 

4.1 Statement of Results 

In this section, we show our main results concerning 
the variational distance criterion, which are proved in 
Sections 4.2 and 4.3 respectively. First, we define the 
quantity 6{Px"Z") as follows. 

Definition 8 Let 1 < M„ < be an integer, and 
Cn ~ {Cz"} z"eZ" be a family of sets such that each 
Cz" C A"" satisfies \Cz"\ = Mn, where \A\ means the 
cardinality of a set A. We define the distribution Pc„ 
on A"" X Z" as 



Pc„(.t",2") = 



Then, we define 



else 



= mind(Px"Z",PcJ, 



(15) 



where the minimization is taken over all possible choice 
of Cn for arbitrary 1 < M„ < [A"!". 



In Section 2.2, we showed that the encoders of com- 
pression limit achieving codes can be used as functions 
for the privacy amplification which is secure in the sense 
of the normalized divergence criterion. However, the 
following Theorem 9 shows a trade-off (with some ex- 
ceptions) between the error probability £($«) and the 
security parameter A(0„) for any code $„. Then, the 
combination of Theorems 9 and 10 states that we can- 
not use the encoders of any (good) Slepian-Wolf codes 
as functions for the secure privacy amplification if we 
employ the variational distance criterion. 

Theorem 9 For arbitrary Slepian-Wolf code $„ = 
i4'n,Tpn), we have 

e($„) + A((/)„) >(5(Px"Z")- 

Theorem 10 If the variance defined in Eq. (10) is 
positive, then we have 

lim (5(Px.>z") = 1. (16) 

n — 'oo 

Corollary 11 For arbitrary Slepian-Wolf code $„ = 

((/)„, l/'n), if 

lim e($„) = 0, 

n — >oo 

then we have 

lim A ((?!)„) = 1 

n — 'oo 

provided that cr^ > 0. 



□ 



From Eq. (5), Corollary 11 means that the actual dis- 
tribution Ps„z" and the ideal distribution Pjj^ x P^-i 
are completely distinguishable asymptotically. 

The combination of Pinsker's inequality and Corol- 
lary 11 imply the following corollary, which states 
that the keys obtained by the encoders of any (good) 
Slepian-Wolf codes do not satisfy the divergence crite- 
rion, although we have shown stronger result in Sec- 
tion 3. The following corollary only suggests that Eve 
might know a few bits about the long key, which is 
not a serious problem in practice. On the other hand, 
the stronger result suggests that Eve's knowledge about 
the key grows infinitely as the length of the key goes to 
infinite, which is a serious problem in practice. 

Corollary 12 For any sequence of Slepian-Wolf codes 
{$„ (0„, t/jn)} such that lim„^oo £($«) = 0, we have 
2 

lim inf £)((/)„) > — - 
?i— too in Z 

provided that cr^ > 0. □ 

Theorems 9 and 10 can be regarded as a general- 
ization of [10, Theorem 4] for the Slepian-Wolf code. 
Therefore, we can also interpret Theorems 9 and 10 as 
follows: The Slepian-Wolf version of the folklore theo- 
rem does not hold for the variational distance criterion. 
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Remark 13 For a distribution Pxz with cr = 0, we 
can easily show that d{Px^Z") = for any n by taking 
Cz^ as the support of Px^i^" (a;"|z"). 

Remark 14 It should be noted that Theorem 9 holds 
not only for i.i.d. random variables (X", Z"), but also 
for any 

4.2 Proof of Theorem 9 

Before we show a proof of Theorem 9, we introduce the 
following lemma. 

Lemma 15 For arbitrary code $„ = there 
exists a code = (</)„,■!/'„) that satisfies 

£«) < (17) 

and 

0„(C(s,;2")) = s V(s,^")eX„xZ". (18) 

Proof. We construct a decoder as follows. If 
(f>n{ipn{s, z")) ^ s, then we set -0^(s,z") = x" for 
arbitrarily chosen G (^~^(s). Otherwise, we set 
iIj',i{s,z^) = ?/'„(s,z"). From the construction of this 
decoder, it is obvious that the code <f>J^ = (0„, tp'^) sat- 
isfies Eqs. (17) and (18). □ 



Proof of Theorem 9 

From Lemma 15, it suffice to prove Theorem 9 for 
codes satisfying Eq. (18). Therefore, we assume that a 
code satisfies Eq. (18) in the rest of this section. 

By using the decoder we construct the map 



,/'„(s,z") = (^.„(s,z"),z"). 



(19) 



Since the decoder satisfies the condition in Eq. (18), ?/;„ 
is an injection map from A^„ x Z" into A"" x Z". 

For the extended code $„ = (0n,V'n)i define 
the error probability 

e(^n) 

= Px"Z"({(x",z") : ^„(0„(x"),z") ^ (a;",z")}). 
Obviously, we have e($n) = e(<&n). 



Next, we define the distribution Pjj^ x P^.! , which 
is the embedding of Pjj^ x P^n into A"" x Z", as follows: 



Py„ X Pz"(x",z") = Pa„ X P2„(^„ (a;",z")) 



for (a;", z") S V„(-A^n x Z"), and Pc/„ x Pz"(a;", z") = 
for other (a;",z"). Similarly, we define the distri- 
bution Ps„Z", which is the embedding of Ps„Z" into 
A-'^xZ"." 

Since the decoder ijjn satisfies Eq. (18), we have 
Px..Z"(?„(s,z")) < Px^zA^",n 



for (s, z") e A^„ X Z", where the equality in Eq. (20) 
follows from Eq. (1) and the definition of Psr^Z"- On 
the other hand, we have 



Px^z- (x", z") > P5,.2" (.t", z") = (21) 

for (a;",z") e (A" x Z")\V^„(7W„ x Z"). Combining 
Eqs. (20) and (21), we have 



d(Px"Z",Ps„Z") 

= Px"Z"((A'" X Z")\^„(X„ X Z")). (22) 

By using Eq. (22), we can rewrite e(<i>„) as 

£(¥„) = Px"Z"((A'" X Z")\V^„(X„ X Z")) 

= diPx'^z^^P^)- 

Finally, from the definition of 5{Px"Z-^) and the 
triangular inequality, we have 



<5(Px"Z") < d{Px'^zr^,Pu„ X P^..) 
< d(Px"Z",/^;:i^) 



+diPs„z'^,Pu„ xPz'^) 
= e($„) + A((^„), 
which completes the proof of Theorem 9. 

4.3 Proof of Theorem 10 



□ 



Let {Cn} be the sequence of the families such that 
d{Px"Z" , Pc„) = S{Px^Z") for each n. For arbitrary 
positive constant 6 > 0, we divide A"" x Z" into the 
following three subsets: 



A, 



{(x",z") : M-^e"^ < P;f„|^„(a:"|z")}, 



A- = {(a;",z") : P^^i^. (.t"|z") < M,; 
and Aa = (A"" x Z")\(yl+ UA-). Let 
Cn= [j {(x",z"): a;"eC.4, 

which is the support of Pc„ . 

We bound 5{Px^z^) as follows: 

<5(Px"Z") 



= P5„Z"(^„(S,Z")) 



(20) 



5] |Px"Z"(x",z")-Pc„(a;",z") 
+ E |Px"Z"(x",z")-Pc„(.t",z")| 

+ E I^'x"Z"(x",z")-Pc„(x",z")| 
> \[{Px^zAA+)-PcM+)) 

+{PcM-) - Px'^zAA-r\C^) 

+Px^zAA-\Cn)) 

+{PcMa)-Px-zAA^))] 

= 1 - (Pc„ (^+) + Px"Z" (^- n c„) 

+Px"Z"(A))- 
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We use the following inequalities 



where is arbitrary real number. 



□ 



< e 



and 



2b 



2Ci 



(23) 
(24) 

(25) 



where Ci is a constant that does not depend on n and 
p is the third moment of —\ogPx\z{X\Z). We will 
prove these inequalities in Appendices B.l, B.2, and 
B.3 respectively. 

From Eqs. (23)-(25) and the fact that 6 > is 
arbitrary, we have 

liminf 5{Px^z^) 



> 1 — lim sup lim sup [Pc„(^+) 



1. 



p 



V"Z' 



■(A)] 



Since the variational distance (divided by 2) is smaller 
than 1 , we have the statement of theorem. □ 

5. Conclusion 

In this paper, we showed that the encoders of (good) 
Slepian-Wolf codes cannot be used as functions for the 
secure privacy amplification in the sense of the varia- 
tional distance criterion nor the divergence criterion. 
The consequence of our result is that we must use the 
privacy amplification not based on the Slepian-Wolf 
code (e.g. [2], [7], [12]. [17]) if we want to employ the cri- 
teria other than weak one (the normalized divergence 
criterion). 
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Appendix A: Proof of Theorem 4 

In order to show a proof of Theorem 4, we need the 
following lemma [13] (see also [8]). 

Lemma 16 For any Slepian-Wolf code $„ = (0„, ■(/'„), 
we have 



e($n) 

1 



log 



p 



.{x'^\z'^) 



For arbitrarily fixed 7 > 0, suppose that there ex- 
ists a code sequence that satisfies Eq. (8) and 

Then, there exists a increasing sequence {riij^x such 
that 

for ever i. 

By using Lemma 16 for «„. 
we have 



nlb + n^H{X\Z), 



1 



b 



log 



1 



> 



(T 



— r-/ni 



n^H{X\Z) 



for every i. By using the central limit theorem, wc have 
hmsupe(<i>„) > hmsupe(<i>„.) 

n — *oo i — >oQ 

>l-G 

which contradict Eq. (8). Therefore, if the code se- 
quence {^n} satisfies Eq. (8), then it satisfies 



lim inf — = log ■ 



^ ^ ^nH[X\Z) 



> 6 - 27. 



Since 7 > is arbitrary, we have the assertion of the 
theorem. □ 

Appendix B 

B.l Proof of Eq. (23) 

From the definitions of Pc„ and ,4+, we have 

< E ^X"Z"(2:",z")e-^v/^ 

B.2 Proof of Eq. (24) 

From the definitions of Pc„ and A- , we have 
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Px-z- (A- n Cn) 



< 



J2 PcAx",z")e"''^' 

(a;",z")e^_nC„ 

B.3 Proof of Eq. (25) 

To simplify the notation, wc introduce the random vari- 
able 



n 



for (X", Z") = ((Xi, Zi), . . . , (X„, Zn))- Then, we can 
rewrite the left hand side of Eq. (25) as 

= Pr{log A/„ - b^/^. < Wn < log Mn + bVTi} 
'\ogMr,-nH{X\Z) b 
a 



Pr 



< 



< 



< 



Pr 



< 



(7^n 
Wn-nH{X\Z) 

\ogMr,-nH{X\Z) b 
a^fn a 
Wn-nH{X\Z) 

\ogMn-nH{X\Z) ^ b 

Wn-nH{X\Z) 

\ogMn-n}i{X\Z) b 
0\fn a 



By using the central limit theorem [16, Corollary 
6], we have 



, Wn-nH{X\Z) ^ \ogM„-nH{X\Z) ^ b 



< 



g{u)du - 



Ci (P 



and 



; Ty„ - nH{X\Z) ^ logM„ - nH{X\Z) , b 



a^fn 



logAf„-nH(X|Z) j, 



a^fn a 



Hence, we have 



Px"Z"(A)< / g(7.)du+^(^) . 

/ iogj\f„-tiH(x|z) b Vcr/ 

Since the interval of the integral is ^ and the height of 
g{u) is lower than 1, we have Eq. (25). 
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